Technologies for hybrid virtualization and secure enclave policy enforcement for edge orchestration

ABSTRACT

Technologies for hybrid virtualization and secure enclave include a computing device and an edge orchestrator. The edge orchestrator securely provisions a container-enclave policy to the computing device. A VMM of the computing device constructs a platform services enclave that includes the container-enclave policy. The platform services enclave requests a local attestation report from an application enclave, and the application enclave generates the attestation report using secure enclave support of a compute engine of the computing device. The attestation report is indicative of a virtualization context of the application enclave, and may include a VM flag, a VMM flag, and a source address of the application enclave. The platform services enclave enforces the container-enclave policy based on the virtualization context of the application enclave. The platform services enclave may control access to functions of the computing device based on the virtualization context. Other embodiments are described and claimed.

RELATED APPLICATIONS

This patent arises from a continuation of U.S. patent application Ser.No. 16/234,731, filed on Dec. 28, 2018, and entitled, “TECHNOLOGIES FORHYBRID VIRTUALIZATION AND SECURE ENCLAVE POLICY ENFORCEMENT FOR EDGEORCHESTRATION.” U.S. patent application Ser. No. 16/234,731 is herebyincorporated by reference in its entirety. Priority to U.S. patentapplication Ser. No. 16/234,731 is hereby claimed.

BACKGROUND

Typical computing devices may use virtualization to provide isolationand multi-tenancy support. Orchestrators may use virtual machines andvirtual network functions as the primary unit of control. Cloudworkloads such as infrastructure as a service (IaaS), platform as aservice (PaaS), and function as a service (FaaS) may rely onvirtualization for isolation and multi-tenancy requirements.

Current processors may provide support for a trusted executionenvironment such as a secure enclave. Secure enclaves include segmentsof memory (including code and/or data) protected by the processor fromunauthorized access including unauthorized reads and writes. Inparticular, certain processors may include Intel® Software GuardExtensions (SGX) to provide secure enclave support. SGX providesconfidentiality, integrity, and replay-protection to the secure enclavedata while the data is resident in the platform memory and thus providesprotection against both software and hardware attacks. The on-chipboundary forms a natural security boundary, where data and code may bestored in plaintext and assumed to be secure. The contents of an SGXsecure enclave may be authenticated and therefore trusted by theindependent software vendor (ISV) that provides the secure enclave.Secure enclaves typically execute in user mode (e.g., ring level 3) andare not aware of platform virtualization context.

BRIEF DESCRIPTION OF THE DRAWINGS

The concepts described herein are illustrated by way of example and notby way of limitation in the accompanying figures. For simplicity andclarity of illustration, elements illustrated in the figures are notnecessarily drawn to scale. Where considered appropriate, referencelabels have been repeated among the figures to indicate corresponding oranalogous elements.

FIG. 1 is a simplified block diagram of at least one embodiment of asystem for hybrid virtualization and secure enclave policy enforcement;

FIG. 2 is a simplified block diagram of at least one embodiment ofvarious environments of the system of FIG. 1;

FIG. 3 is a simplified flow diagram of at least one embodiment of amethod for hybrid virtualization and secure enclave policy enforcementthat may be executed by a computing device of FIGS. 1-2; and

FIG. 4 is a simplified block diagram of at least one embodiment of anedge architecture that may include the system of FIGS. 1-2.

DETAILED DESCRIPTION OF THE DRAWINGS

While the concepts of the present disclosure are susceptible to variousmodifications and alternative forms, specific embodiments thereof havebeen shown by way of example in the drawings and will be describedherein in detail. It should be understood, however, that there is nointent to limit the concepts of the present disclosure to the particularforms disclosed, but on the contrary, the intention is to cover allmodifications, equivalents, and alternatives consistent with the presentdisclosure and the appended claims.

References in the specification to “one embodiment,” “an embodiment,”“an illustrative embodiment,” etc., indicate that the embodimentdescribed may include a particular feature, structure, orcharacteristic, but every embodiment may or may not necessarily includethat particular feature, structure, or characteristic. Moreover, suchphrases are not necessarily referring to the same embodiment. Further,when a particular feature, structure, or characteristic is described inconnection with an embodiment, it is submitted that it is within theknowledge of one skilled in the art to effect such feature, structure,or characteristic in connection with other embodiments whether or notexplicitly described. Additionally, it should be appreciated that itemsincluded in a list in the form of “at least one A, B, and C” can mean(A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C).Similarly, items listed in the form of “at least one of A, B, or C” canmean (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C).

The disclosed embodiments may be implemented, in some cases, inhardware, firmware, software, or any combination thereof. The disclosedembodiments may also be implemented as instructions carried by or storedon a transitory or non-transitory machine-readable (e.g.,computer-readable) storage medium, which may be read and executed by oneor more processors. Furthermore, the disclosed embodiments may beinitially encoded as a set of preliminary instructions (e.g., encoded ona machine-readable storage medium) that may require a preliminaryprocessing operations to prepare the instructions for execution on adestination device. The preliminary processing may include combining theinstructions with data present on a device, translating the instructionsto a different format, performing compression, decompression,encryption, and/or decryption, combining multiple files that includedifferent sections of the instructions, integrating the instructionswith other code present on a device, such as a library, an operatingsystem, etc., or similar operations. The preliminary processing may beperformed by the source compute device (e.g., the device that is to sendthe instructions), the destination compute device (e.g., the device thatis to execute the instructions), or an intermediary device. Amachine-readable storage medium may be embodied as any storage device,mechanism, or other physical structure for storing or transmittinginformation in a form readable by a machine (e.g., a volatile ornon-volatile memory, a media disc, or other media device).

In the drawings, some structural or method features may be shown inspecific arrangements and/or orderings. However, it should beappreciated that such specific arrangements and/or orderings may not berequired. Rather, in some embodiments, such features may be arranged ina different manner and/or order than shown in the illustrative figures.Additionally, the inclusion of a structural or method feature in aparticular figure is not meant to imply that such feature is required inall embodiments and, in some embodiments, may not be included or may becombined with other features.

Referring now to FIG. 1, a system 100 for hybrid virtualization andsecure enclave policy enforcement includes a computing device 102 andedge orchestrator 104. In use, as described further below, the edgeorchestrator 104 may securely provision a container-enclave policy (CEP)to the computing device 102. An enforcement component of the computingdevice 102 such as a platform services enclave (PSE) receivesattestation reports from application enclaves executed by the computingdevice 102. The attestation reports are indicative of the virtualizationcontext of each application enclave. The PSE enforces the CEP based onthe virtualization context of each application enclave. Thus, the system100 may differentiate between enclaves that are expected to originatefrom containers under the control of a VMM from those that are not,which may prevent attacks from rogue enclaves or other maliciousprocesses. Additionally, by allowing the attestation report to indicatevirtualization context, the system 100 does not require the trusted codebase (TCB) of the enclave to be expanded to include pre-boot firmware,VM, or VMM components.

The computing device 102 may be embodied as any type of device capableof performing the functions described herein. For example, the computingdevice 102 may be embodied as, without limitation, a switch, a router, anetwork device, a computer, a mobile computing device, a server, aworkstation, a multiprocessor system, a distributed computing device,and/or a consumer electronic device. Additionally or alternatively, thecomputing device 102 may be embodied as a one or more compute sleds,memory sleds, or other racks, sleds, computing chassis, or othercomponents of a physically disaggregated computing device. As shown inFIG. 1, the illustrative computing device 102 includes a compute engine120, an I/O subsystem 126, a memory 128, a data storage device 130, anda communication subsystem 132. Additionally, in some embodiments, one ormore of the illustrative components may be incorporated in, or otherwiseform a portion of, another component. For example, the memory 128, orportions thereof, may be incorporated in the compute engine 120 in someembodiments.

The compute engine 120 may be embodied as any type of compute enginecapable of performing the functions described herein. For example, thecompute engine 120 may be embodied as a single or multi-coreprocessor(s), digital signal processor, microcontroller,field-programmable gate array (FPGA), or other configurable circuitry,application-specific integrated circuit (ASIC), or other processor orprocessing/controlling circuit. The compute engine 120 includes VMXsupport 122. The VMX support 122 supports virtualized execution ofoperating systems by providing two modes of execution: VMX root mode andVMX non-root mode. The VMX root mode allows executing software to havebroad control of the computing device 102 and its hardware resources.Accordingly, a virtual machine monitor (VMM), hypervisor, or hostoperating system may execute in VMX root mode. The VMX non-root moderestricts access to certain hardware instructions while stillimplementing the ordinary ring/privilege system of the compute engine120. Thus, one or more guest virtual machines (VMs) and/or guestoperating systems (OSs) may execute in the VMX non-root mode. Thoseguest OSs may execute in ring zero, similar to execution withoutvirtualization. The execution of certain hardware instructions andcertain other system events may trigger hardware-assisted transitions toVMX root mode. Those hardware-assisted transitions are commonly known asvirtual machine exits (VM exits) or hypercalls. Upon encountering a VMexit, the compute engine 120 may switch context from the guest VM to theVMM in order to handle the VM exit. The VMX support 122 may be embodiedas, for example, Intel VT-x technology and/or Intel VT-d technology.

The compute engine 120 also includes secure enclave support 124, whichallows the compute engine 120 to establish a trusted executionenvironment known as a secure enclave, in which executing code may bemeasured, verified, and/or otherwise determined to be authentic.Additionally, code and data included in the secure enclave may beencrypted or otherwise protected from being accessed by code executingoutside of the secure enclave. For example, code and data included inthe secure enclave may be protected by hardware protection mechanisms ofthe compute engine 120 while being executed or while being stored incertain protected cache memory of the compute engine 120. The code anddata included in the secure enclave may be encrypted when stored in ashared cache or the main memory 128. The secure enclave support 124 maybe embodied as a set of processor instruction extensions that allows thecompute engine 120 to establish one or more secure enclaves in thememory 128. For example, the secure enclave support 124 may be embodiedas Intel Software Guard Extensions (SGX) technology.

The memory 128 may be embodied as any type of volatile or non-volatilememory or data storage capable of performing the functions describedherein. In operation, the memory 128 may store various data and softwareused during operation of the computing device 102 such as operatingsystems, applications, programs, libraries, and drivers. As shown, thememory 128 may be communicatively coupled to the compute engine 120 viathe I/O subsystem 126, which may be embodied as circuitry and/orcomponents to facilitate input/output operations with the compute engine120, the memory 128, and other components of the computing device 102.For example, the I/O subsystem 126 may be embodied as, or otherwiseinclude, memory controller hubs, input/output control hubs, sensor hubs,host controllers, firmware devices, communication links (i.e.,point-to-point links, bus links, wires, cables, light guides, printedcircuit board traces, etc.) and/or other components and subsystems tofacilitate the input/output operations. In some embodiments, the memory128 may be directly coupled to the compute engine 120, for example viaan integrated memory controller hub. Additionally, in some embodiments,the I/O subsystem 126 may form a portion of a system-on-a-chip (SoC) andbe incorporated, along with the compute engine 120, the memory 128,and/or other components of the computing device 102, on a singleintegrated circuit chip.

The data storage device 130 may be embodied as any type of device ordevices configured for short-term or long-term storage of data such as,for example, memory devices and circuits, memory cards, hard diskdrives, solid-state drives, non-volatile flash memory, or other datastorage devices. The communications subsystem 132 may be embodied as anycommunication circuit, device, or collection thereof, capable ofenabling communications between the computing device 102 and otherremote devices over the network 106. The communications subsystem 132may be configured to use any one or more communication technology (e.g.,wired or wireless communications) and associated protocols (e.g.,Ethernet, Bluetooth®, Wi-Fi®, WiMAX, 3G, 4G LTE, 5G, etc.) to effectsuch communication.

The manageability engine 134 may be embodied as any hardwarecomponent(s) or circuitry capable of providing manageability andsecurity-related services to the computing device 102. In particular,the manageability engine 134 may include a microprocessor,microcontroller, or other embedded controller capable of executingfirmware and/or other code independently and securely from the computeengine 120. For example, the manageability engine 134 may be embodied asa manageability engine (ME), a converged security and manageabilityengine (CSME), an Intel® innovation engine (IE), a board managementcontroller (BMC), an embedded controller (EC), a trusted hardwarecomponent such as a trusted platform module (TPM), a hardware securemodule (HSM), or other independent controller of the computing device102. In some embodiments, the manageability engine 134 may include a TPMimplemented with secure firmware, such as Intel Platform TrustTechnology (PTT). Thus, the manageability engine 134 may be used toestablish a trusted execution environment for the computing device 102.The manageability engine 134 may communicate with the compute engine 120and/or other components of the computing device 102 over an I/O linksuch as PCI Express or over a dedicated bus, such as a host embeddedcontroller interface (HECI). Further, in some embodiments, themanageability engine 134 is also capable of communicating using thecommunication subsystem 132 or a dedicated communication circuitindependently of the state of the computing device 102 (e.g.,independently of the state of the main compute engine 120), al so knownas “out-of-band” communication.

The edge orchestrator 104 may be embodied as any type of computation orcomputer device capable of performing the functions described herein,including, without limitation, a switch, a router, a network device, acomputer, a mobile computing device, a server, a workstation, amultiprocessor system, a distributed computing device, and/or a consumerelectronic device. Additionally or alternatively, the edge orchestrator104 may be embodied as a one or more compute sleds, memory sleds, orother racks, sleds, computing chassis, or other components of aphysically disaggregated computing device. As such, the edgeorchestrator 104 may include components and features similar to thecomputing device 102, such as a compute engine 120, I/O subsystem 126,memory 128, data storage 130, communication subsystem 132, and/orvarious peripheral devices. Those individual components of the edgeorchestrator 104 may be similar to the corresponding components of thecomputing device 102, the description of which is applicable to thecorresponding components of the edge orchestrator 104 and is notrepeated for clarity of the present description.

As discussed in more detail below, the computing device 102 and the edgeorchestrator 104 may be configured to transmit and receive data witheach other and/or other devices of the system 100 over the network 106.The network 106 may be embodied as any number of various wired and/orwireless networks. For example, the network 106 may be embodied as, orotherwise include a mobile access network, a network edgeinfrastructure, a wired or wireless local area network (LAN), and/or awired or wireless wide area network (WAN). As such, the network 106 mayinclude any number of additional devices, such as additional basestations, access points, computers, routers, and switches, to facilitatecommunications among the devices of the system 100. In the illustrativeembodiment, the network 106 is embodied as an edge network fabric.

Referring now to FIG. 2, in an illustrative embodiment, the computingdevice 102 establishes an environment 200 during operation. Theillustrative environment 200 includes virtual machines (VMs) 202,application enclaves 204, a virtual machine monitor (VMM) 206, aplatform services enclave (PSE) 208, and a policy manager 214. Thevarious components of the environment 200 may be embodied as hardware,firmware, software, or a combination thereof As such, in someembodiments, one or more of the components of the environment 200 may beembodied as circuitry or collection of electrical devices such as VMcircuitry 202, application enclave circuitry 204, VMM circuitry 206, PSEcircuitry 208, and/or policy manager circuitry 214). It should beappreciated that, in such embodiments, one or more of the VM circuitry202, the application enclave circuitry 204, the VMM circuitry 206, thePSE circuitry 208, and/or the policy manager circuitry 214 may form aportion of the compute engine 120, the I/O subsystem 126, themanageability engine 134, and/or other components of the computingdevice 102. Additionally, in some embodiments, one or more of theillustrative components may form a portion of another component and/orone or more of the illustrative components may be independent of oneanother.

The virtual machine 202 may be embodied as a VM, virtual networkfunction (VNF), service VM (SVM), or other virtualized workload that maybe executed by the computing device 102. Similarly, the VMM 206 may beembodied as any virtual machine monitor, hypervisor, or other componentthat manages execution of one or more virtualized workloads on thecomputing device 102. The VMM 206 may manage virtualized workloads usingthe VMX support 122 of the compute engine 120, for example by executingin a VMX root mode or other privileged operating mode. Additionally,although illustrated as including a single VM 202, it should beunderstood that the environment 200 may include multiple VMs 202.

Each application enclave 204 may be embodied as an SGX secure enclave orother trusted execution environment that may be executed by thecomputing device 102. Each application enclave 204 executes within auser process or other application process executed by the computingdevice 102. As shown, certain application enclaves 204 may executewithin a VM 202, and certain application enclaves 204 may executeoutside of a VM 202 (e.g., without virtualization). Thus, eachapplication enclave 204 executes in a non-privileged mode, such as VMXnon-root mode, guest mode, user mode, ring level 3, or othernon-privileged mode.

Similarly, the PSE 208 may be embodied as an SGX secure enclave or othertrusted execution environment that may be executed by the computingdevice 102. The PSE 208 executes in a non-privileged mode, such as VMXnon-root mode, guest mode, user mode, ring level 3, or othernon-privileged mode. However, unlike the application enclaves 204, thePSE 208 may be embodied as an architectural enclave or other enclavethat is privileged by the compute engine 120 and/or the VMM 206 toaccess certain hardware resources of the computing device 102, such asthe manageability engine 134.

The policy manager 214 is configured to securely provision acontainer-enclave policy 212 to the computing device 102. Securelyprovisioning the container-enclave policy 212 may include receiving thepolicy 212 by the manageability engine 134 from an external edgeorchestrator 104. The policy manager 214 may be further configured toinject the container enclave policy 212 into an image of the VMM 206.

The VMM 206 is configured to construct the PSE 208 in response toprovisioning the container-enclave policy 212. As shown, the PSE 208includes the container-enclave policy 212. In some embodiments,constructing the PSE 208 may include loading the container-enclavepolicy 212 with hardware I/O virtualization protection (e.g., the VMX122 of the compute engine 120). In some embodiments, constructing thePSE 208 may include securely provisioning the container-enclave policy212 to the PSE 208 with a provisioning key. The VMM 206 may be furtherconfigured to maintain a list of VM source addresses associated with VMs202 that are successfully loaded by the computing device 102.

The PSE 208 is configured to request a local attestation report from anapplication enclave 204. The local attestation report is indicative of avirtualization context of the application enclave 204. For example, theattestation report may include one or more flags indicative of whetherthe application enclave 204 is launched within a VM 202 or within a VMM206. The attestation report may include a source address of a creatingprocess of the application enclave 204, which may be a memory pageoffset of an ECREATE processor instruction associated with theapplication enclave 204. The PSE 208 is further configured to update avirtualization context table 210 as a function of the local attestationreport received from the application enclave 204. The PSE 208 is furtherconfigured to enforce the container-enclave policy 212 based on thevirtualization context of the application enclave 204 (e.g., using thevirtualization context table 210).

Enforcing the container-enclave policy 212 may include controllingaccess to a local function of the computing device 102 or controllingaccess to a hardware function of a hardware security module of thecomputing device 102 (e.g., the manageability engine 134) based on thevirtualization context of the application enclave 204. Enforcing thecontainer-enclave policy 212 may include determining whether theapplication enclave 204 is launched within a VMM 206 or within a VM 202.Enforcing the container-enclave policy 212 may include comparing asource address of a creating process of the application enclave 204 to apredetermined source address associated with the application enclave204, such as a VM source address of a VM 202 that has been successfullyloaded.

The application enclave 204 is configured to generate the attestationreport in response to the request from the PSE 208. To generate theattestation report, the application enclave 204 is configured to invokethe secure enclave support 124 of the compute engine 120, and, inresponse, the compute engine 120 generates the attestation report. Forexample, the application enclave 204 may invoke an EREPORT processorinstruction of the compute engine 120.

Referring now to FIG. 3, in use, the computing device 102 may execute amethod 300 for hybrid virtualization and secure enclave policyenforcement. It should be appreciated that, in some embodiments, theoperations of the method 300 may be performed by one or more componentsof the environment 200 of the computing device 102 as shown in FIG. 2.The method 300 begins in block 302, in which the computing device 102securely provisions the container-enclave policy 212 to the computingdevice 102. For example, the computing device 102 may securely receivethe container-enclave policy 212 from the edge orchestrator 104 andstore the container-enclave policy 212 in secure storage accessible tothe compute engine 120, chipset, or other components of the computingdevice 102. In some embodiments, the manageability engine 134 mayreceive the container-enclave policy 212 from the edge orchestratorusing out-of-band communication. In some embodiments, in block 304 thecomputing device 102 may store the container-enclave policy 212 with asecure hardware module such as the manageability engine 134 (e.g., a TPMor PTT). The container-enclave policy 212 may be stored in securenonvolatile storage that is accessible to or otherwise managed by themanageability engine 134. In some embodiments, in block 306 themanageability engine 134 may inject the container-enclave policy 212securely into an image of the VMM 206. For example, thecontainer-enclave policy 212 may be inserted into a binary image of theVMM 206 stored in the data storage 130, so that the container-enclavepolicy 212 will be loaded with the VMM 206 image.

In block 308, the computing device 102 loads and executes the VMM 206.For example, the computing device 102 may load the VMM 206 in responseto booting or another system start event. In block 310 the computingdevice 102 constructs the platform services enclave (PSE) 208 within theVMM 206. As described above, the PSE 208 may be constructed within anon-privileged mode managed by the VMM 206, such as ring level 3. Forexample, the VMM 206 may invoke one or more processor instructions withthe secure enclave support 124 of the compute engine 120 to load memorypages into a secure enclave address range associated with the PSE 208(e.g., ECREATE, EADD, or other instructions). Loading memory pages mayupdate an enclave measurement of the PSE 208. After loading the PSE 208into memory, the VMM 206 may invoke one or more processor instructionsto finalize the enclave measurement and initialize the PSE 208 forexecution (e.g., EINIT). The VMM 206 may cause the PSE 208 to startexecution, for example by invoking one or more processor instructions toenter the PSE 208 (e.g., EENTER).

After construction and/or initialization, the PSE 208 includes thecontainer-enclave policy 212. In some embodiments, in block 312 thecomputing device 102 may securely load the container-enclave policy 212with hardware I/O virtualization protection, such as Intel VT-d directedI/O protection. For example, the container-enclave policy 212 may beloaded from the manageability engine 134 and/or from another I/O deviceof the computing device 102. In some embodiments, in block 314 thecontainer-enclave policy 212 may be securely provisioned to the PSE 208as it is executing. The container-enclave policy 212 may be protectedwith a provisioning key managed by the secure enclave support 124 of thecompute engine 120.

In block 316, the VMM 206 and/or the PSE 208 maintains a list of sourceaddresses for successfully loaded VMs 202. The list may include, forexample, memory page offsets included in each VM 202. As describedfurther below, the list of memory page offsets may be used to identifywhich VM 202 includes a particular application enclave 204.

In block 318, the PSE 208 requests a local attestation report from anapplication enclave 204 executed by the computing device 102. Asdescribed further below, the attestation report includes informationdescribing the virtualization context of the application enclave 204.The PSE 208 may request the attestation report as each applicationenclave 204 is launched, in response to an application enclave 204requesting services from the VMM 206 and/or the PSE 208, periodicallyduring execution of each application enclave 204, and/or in response toother policy enforcement events.

In block 320, the application enclave 204 generates the localattestation report using the secure enclave support 124 of the computeengine 120. For example, the application enclave 204 may invoke one ormore processor instructions with the secure enclave support 124 of thecompute engine 120 to generate the report (e.g., EREPORT). The reportincludes data indicative of the virtualization context of theapplication enclave 204, including whether or not the applicationenclave 204 is executing within a VM 202 or a VMM 206 and including asource address of a process that created the application enclave 204.The virtualization context data may be added to the report and signed bythe compute engine 120, which may prevent malicious software fromgenerating false attestation reports. Virtualization context data may bemade available through microcode, xucode instructions, or throughmicroarchitecture circuitry of the compute engine 120. Accordingly, thevirtualization context data may not be falsified or tampered with bysoftware such as the application enclave 204, the VMM 206 and/or the VMs202. In some embodiments, in block 322, the attestation report mayinclude an attributes field having flags that indicate the VM/VMMvirtualization context of the application enclave 204. For example, theattributes may include a VM flag that is set if the application enclave204 is executing within a VM 202 and that is cleared if the applicationenclave 204 is not executing within a VM 202. Similarly, the attributesmay include a VMM flag that is set if the application enclave 204 isexecuting within a VMM 206 and that is cleared if the applicationenclave 204 is not executing within a VMM 206. One potential embodimentof an attributes field that may be included in the report is shown belowin Table 2. In some embodiments, in block 324, the attestation reportmay include a source address field that indicates the source address ofa process that created the application enclave 204. For example, thesource address may include the address of the process from which theECREATE instruction was invoked to create the application enclave 204.In some embodiments, the source address field may include a part of thecomplete address, such as a memory page offset, page number, segmentaddress, or other part of the address. One potential embodiment of areport structure including the source address field is shown below inTable 1.

TABLE 1 Attestation REPORT structure. Offset Size Field (B) (B)Description CPUSVN 0 16 The security version number of the processor.MISCSELECT 16 4 SSA Frame specified extended feature set bit vectorRESERVED 20 28 Must be zero ATTRIBUTES 48 16 The values of theATTRIBUTES flags for the enclave. MRENCLAVE 64 32 The value of SECS.MRENCLAVE RESERVED 96 32 Reserved MRSIGNER 128 32 The value ofSECS.MRSIGNER RESERVED 160 96 Zero ISVPRODID 256 02 Enclave PRODUCT IDISVSVN 258 02 The security version number of the Enclave RESERVED 260 60Zero REPORTDATA 320 64 A set of data used for communication between theenclave and the target enclave. KEYID 384 32 Value for key wear-outprotection SRCADDR 416 8 Page address where ECREATE was called MAC 42416 The CMAC on the report using report key

TABLE 2 ATTRIBUTES field of REPORT structure. Field Bit PositionDescription RESERVED  0 DEBUG  1 If 1, the enclave permit debugger toread and write data to enclave MODE64BIT  2 Enclave runs in 64-bit modeRESERVED  3 Must be Zero PRO VISIONKEY  4 Provisioning Key is availablefrom EGETKEY EINITTOKENKEY  5 EINIT token key is available from EGETKEYVMM-bit  6 Enclave has VMM context VM-bit  7 Enclave has VM contextRESERVED  63:8 XFRM 127:64 XSAVE Feature Request Mask.

In block 326, the PSE 208 updates the virtualization context table 210based on the report received from the application enclave 204. Ofcourse, the PSE 208 may verify the local attestation report beforeupdating the virtualization context table 210, for example by verifyinga signature, message authentication code, or other verification dataincluded in the report. The virtualization context table 210 may includethe VM/VMM virtualization context, the source address, and/or othervirtualization context data for each application enclave 204 executed bythe computing device 102.

In block 328, the PSE 208 enforces the container-enclave policy 212based on the virtualization context of the application enclave 204. ThePSE 208 may allow or deny access to platform features, local functions,and other operations based on the virtualization context. The PSE 208may instruct or otherwise cause the VMM 206 to take appropriate policyenforcement actions. The policy determination may be based on the VM/VMMcontext of the application enclave 204 and/or the identity of thecontaining process of the application enclave 204. Each applicationenclave 204 may be associated with a different container-enclave policy212. In some embodiments, in block 330 the PSE 208 may allow or denyaccess to one or more functions based on the virtualization context. Thefunctions may be, for example, local functions of the applicationenclave 204 or its containing process, functions provided by the VMM 206or other supervisor process of the computing device 102, or any otherlocal process. In some embodiments, the functions may be embodied assecurity services or other hardware functions provided by themanageability engine 134 or other secure hardware component. Thecontainer-enclave policy 212 may identify functions, for example, byname, address, or other identifier. In some embodiments, in block 332,the PSE 208 may evaluate the container-enclave policy 212 based on theVM flag and/or the VMM flag of the attestation report. Differentfunctions may be allowed or denied based on the virtualization context.For example, an application enclave 204 may be allowed to invoke certainlocal functions when executing within a VM 202 (VM flag set) but notallowed to invoke those functions when executing within a VMM 206 (VMMflag set). As another example, an application enclave 204 may be allowedto access certain functions when executed in a process withoutvirtualization (VM flag cleared and VMM flag cleared). In someembodiments, in block 334, the PSE 208 may evaluate thecontainer-enclave policy 212 based on the source address of theapplication enclave 204. The source address may be compared to apredetermined list of allowed addresses, which may be associated withprocesses or VMs 202. For example, an application enclave 204 may beallowed or denied to execute only within a particular user process or VM202. Continuing that example, an application enclave 204 may be allowedto access certain functions (e.g., functions of the manageability engine134 or other hardware functions) when executed within a service VM 202or other predetermined VM 202 of the computing device 102. Afterenforcing the container-enclave policy 212, the method 300 loops back toblock 316 to continue monitoring and enforcing the container-enclavepolicy 212 for the application enclaves 204.

Referring now to FIG. 4, diagram 400 shows an edge architecture that mayinclude the system 100. As shown, the edge architecture includesmultiple layers 402, 404, 406, 408. Each layer includes multiple nodesthat may communicate with an edge fabric to other nodes of the samelayer and/or nodes at other layers. Instances of the computing device102 may be included at one or more different layers 402, 404, 406, 408.For example, the computing device 102 may be embodied as an edge nodeserver, edge gateway, endpoint device, or other device in the edgearchitecture. The things/endpoint layer 402 may include large numbers ofendpoint devices (e.g., computing devices 102) that are heterogeneous,may be mobile, and are widely distributed geographically. Theaccess/edge layer 404 may include access network components such aswireless towers, access points, base stations, intermediate nodes,gateways, fog nodes, central offices, and other access network or edgecomponents. As described above, the access/edge layer 404 may includethe computing device 102 and/or the edge orchestrator 104. Components ofthe access/edge layer 404 may be distributed at the building, smallcell, neighborhood, or cell scale. Thus, components of the access/edgelayer 404 may be relatively close in physical proximity to components ofthe things/endpoint layer 402. The core network layer 406 may includecore network routers, network gateways, servers, and othermore-centralized computing devices. Components of the core network layer406 may be distributed regionally or nationally. The cloud/Internetlayer 408 may include Internet backbone routers, cloud serviceproviders, datacenters, and other cloud resources. The components of thecloud/Internet layer 408 may be distributed globally.

As shown, the edge architecture is organized according to a logicalgradient 410 from global, cloud-based components toward local, endpointdevices. Components that are closer to the network edge (i.e., closer tothe endpoint layer 402) may be smaller but more numerous, with fewerprocessing resources and lower power consumption, as compared tocomponents that are closer to the network core (i.e., closer to thecloud/Internet layer 408). However, network communications amongcomponents closer to the network edge may be faster and/or have lowerlatency as compared to communications that traverse through layerscloser to the network core. The same logical gradient 410 may apply tocomponents within a layer. For example, the access/edge layer 404 mayinclude numerous, widely spread base stations, street cabinets, andother access nodes as well as less-numerous but more sophisticatedcentral offices or other aggregation nodes. Thus, by including secureorchestration in the access/edge layer 404 or other components close tothe network edge, the system 100 may improve latency and performance ascompared to traditional cloud-computing architectures.

It should be appreciated that, in some embodiments, the method 300 maybe embodied as various instructions stored on a computer-readable media,which may be executed by the compute engine 120, the I/O subsystem 126,the manageability engine 134, and/or other components of the computingdevice 102 to cause the computing device 102 to perform the method 300.The computer-readable media may be embodied as any type of media capableof being read by the computing device 102 including, but not limited to,the memory 128, the data storage device 130, firmware devices, othermemory or data storage devices of the computing device 102, portablemedia readable by a peripheral device of the computing device 102,and/or other media.

EXAMPLES

Illustrative examples of the technologies disclosed herein are providedbelow. An embodiment of the technologies may include any one or more,and any combination of, the examples described below.

Example 1 includes a computing device for secure virtualization, thecomputing device comprising: a processor with secure enclave support andvirtualization support; and a platform services enclave to request alocal attestation report from an application enclave of the computingdevice; wherein the processor is to generate, with the secure enclavesupport and in response to an invocation by the application enclave, anattestation report, wherein the attestation report is indicative of avirtualization context of the application enclave, wherein thevirtualization context is maintained by the virtualization support ofthe processor; and wherein the platform services enclave is to enforce acontainer-enclave policy based on the virtualization context of theapplication enclave.

Example 2 includes the subject matter of Example 1, and wherein theapplication enclave is to invoke an EREPORT processor instruction of theprocessor.

Example 3 includes the subject matter of any of Examples 1 and 2, andwherein the attestation report includes a flag indicative of whether theapplication enclave is launched within a virtual machine monitor.

Example 4 includes the subject matter of any of Examples 1-3, andwherein the attestation report includes a flag indicative of whether theapplication enclave is launched within a virtual machine.

Example 5 includes the subject matter of any of Examples 1-4, andwherein the attestation report includes a source address of a creatingprocess of the application enclave.

Example 6 includes the subject matter of any of Examples 1-5, andwherein the source address comprises a memory page offset of a processorinstruction that creates the application enclave.

Example 7 includes the subject matter of any of Examples 1-6, andwherein the processor instruction comprises an ECREATE processorinstruction.

Example 8 includes the subject matter of any of Examples 1-7, andwherein the platform services enclave is further to update avirtualization context table as a function of the local attestationreport.

Example 9 includes the subject matter of any of Examples 1-8, andwherein to enforce the container-enclave policy comprises to controlaccess to a local function of the computing device based on thevirtualization context of the application enclave.

Example 10 includes the subject matter of any of Examples 1-9, andwherein to enforce the container-enclave policy comprises to controlaccess to a hardware function of a hardware security module of thecomputing device based on the virtualization context of the applicationenclave.

Example 11 includes the subject matter of any of Examples 1-10, andwherein to enforce the container-enclave policy comprises to determinewhether the application enclave is launched within a virtual machinemonitor or whether the application enclave is launched within a virtualmachine.

Example 12 includes the subject matter of any of Examples 1-11, andwherein to enforce the container-enclave policy comprises to compare asource address of a creating process of the application enclave to apredetermined source address associated with the application enclave.

Example 13 includes the subject matter of any of Examples 1-12, andfurther comprising a virtual machine monitor to maintain a list ofvirtual machine source addresses associated with virtual machines thatare successfully loaded by the computing device, wherein thepredetermined source address is included in the list of virtual machinesource addresses.

Example 14 includes the subject matter of any of Examples 1-13, andfurther comprising: a policy manager to securely provision thecontainer-enclave policy to the computing device; and a virtual machinemonitor to construct the platform services enclave in response to secureprovisioning of the container-enclave policy, wherein the platformservice enclave includes the container-enclave policy.

Example 15 includes the subject matter of any of Examples 1-14, andfurther comprising a manageability engine, wherein to securely provisionthe container-enclave policy comprises to receive, by the manageabilityengine, the container-enclave policy from an external orchestrator.

Example 16 includes the subject matter of any of Examples 1-15, andwherein the manageability engine is further to inject the containerenclave policy into an image of the virtual machine monitor.

Example 17 includes the subject matter of any of Examples 1-16, andwherein to construct the platform services enclave comprises to load thecontainer-enclave policy with hardware I/O virtualization protection.

Example 18 includes the subject matter of any of Examples 1-17, andwherein to construct the platform services enclave comprises to securelyprovision the container-enclave policy to the platform services enclavewith a provisioning key.

Example 19 includes a method for secure virtualization, the methodcomprising: requesting, by a platform services enclave of a computingdevice, a local attestation report from an application enclave of thecomputing device; generating, by the application enclave, an attestationreport using secure enclave support of a processor of the computingdevice, wherein the attestation report is indicative of a virtualizationcontext of the application enclave, wherein the virtualization contextis provided by virtualization support of the processor; and enforcing,by the platform services enclave, a container-enclave policy based onthe virtualization context of the application enclave.

Example 20 includes the subject matter of Example 19, and whereingenerating the attestation report comprises invoking an EREPORTprocessor instruction of the processor.

Example 21 includes the subject matter of any of Examples 19 and 20, andwherein the attestation report includes a flag indicative of whether theapplication enclave is launched within a virtual machine monitor.

Example 22 includes the subject matter of any of Examples 19-21, andwherein the attestation report includes a flag indicative of whether theapplication enclave is launched within a virtual machine.

Example 23 includes the subject matter of any of Examples 19-22, andwherein the attestation report includes a source address of a creatingprocess of the application enclave.

Example 24 includes the subject matter of any of Examples 19-23, andwherein the source address comprises a memory page offset of processorinstruction that creates the application enclave.

Example 25 includes the subject matter of any of Examples 19-24, andwherein the processor instruction comprises an ECREATE processorinstruction.

Example 26 includes the subject matter of any of Examples 19-25, andfurther comprising updating, by the platform services enclave, avirtualization context table as a function of the local attestationreport.

Example 27 includes the subject matter of any of Examples 19-26, andwherein enforcing the container-enclave policy comprises controllingaccess to a local function of the computing device based on thevirtualization context of the application enclave.

Example 28 includes the subject matter of any of Examples 19-27, andwherein enforcing the container-enclave policy comprises controllingaccess to a hardware function of a hardware security module of thecomputing device based on the virtualization context of the applicationenclave.

Example 29 includes the subject matter of any of Examples 19-28, andwherein enforcing the container-enclave policy comprises determiningwhether the application enclave is launched within a virtual machinemonitor or whether the application enclave is launched within a virtualmachine.

Example 30 includes the subject matter of any of Examples 19-29, andwherein enforcing the container-enclave policy comprises comparing asource address of a creating process of the application enclave to apredetermined source address associated with the application enclave.

Example 31 includes the subject matter of any of Examples 19-30, andfurther comprising maintaining, by the computing device, a list ofvirtual machine source addresses associated with virtual machines thatare successfully loaded by the computing device, wherein thepredetermined source address is included in the list of virtual machinesource addresses.

Example 32 includes the subject matter of any of Examples 19-31, andfurther comprising: securely provisioning, by the computing device, thecontainer-enclave policy to the computing device; and constructing, bythe computing device, the platform services enclave by a virtual machinemonitor of the computing device in response to securely provisioning thecontainer-enclave policy, wherein the platform service enclave includesthe container-enclave policy.

Example 33 includes the subject matter of any of Examples 19-32, andwherein securely provisioning the container-enclave policy comprisesreceiving, by a manageability engine of the computing device, thecontainer-enclave policy from an external orchestrator.

Example 34 includes the subject matter of any of Examples 19-33, andfurther comprising injecting, by the manageability engine, the containerenclave policy into an image of the virtual machine monitor.

Example 35 includes the subject matter of any of Examples 19-34, andwherein constructing the platform services enclave comprises loading thecontainer-enclave policy with hardware I/O virtualization protection.

Example 36 includes the subject matter of any of Examples 19-35, andwherein constructing the platform services enclave comprises securelyprovisioning the container-enclave policy to the platform servicesenclave with a provisioning key.

Example 37 includes a computing device comprising: a processor; and amemory having stored therein a plurality of instructions that whenexecuted by the processor cause the computing device to perform themethod of any of Examples 19-36.

Example 38 includes one or more non-transitory, computer-readablestorage media comprising a plurality of instructions stored thereon thatin response to being prepared for execution and subsequently beingexecuted result in a computing performing the method of any of Examples19-36.

Example 39 includes a computing device comprising means for performingthe method of any of Examples 19-36.

1. A computing device for secure virtualization, the computing devicecomprising: a processor with secure enclave support and virtualizationsupport; and a platform services enclave to request a local attestationreport from an application enclave of the computing device; wherein theprocessor is to generate, with the secure enclave support and inresponse to an invocation by the application enclave, an attestationreport, wherein the attestation report is indicative of a virtualizationcontext of the application enclave, wherein the virtualization contextis provided by the virtualization support of the processor; and whereinthe platform services enclave is to enforce a container-enclave policybased on the virtualization context of the application enclave.
 2. Thecomputing device of claim 1, wherein the attestation report includes aflag indicative of whether the application enclave is launched within avirtual machine monitor.
 3. The computing device of claim 1, wherein theattestation report includes a flag indicative of whether the applicationenclave is launched within a virtual machine.
 4. The computing device ofclaim 1, wherein the attestation report includes a source address of acreating process of the application enclave.
 5. The computing device ofclaim 4, wherein the source address comprises a memory page offset of aprocessor instruction that creates the application enclave.
 6. Thecomputing device of claim 1, wherein to enforce the container-enclavepolicy comprises to control access to a local function of the computingdevice based on the virtualization context of the application enclave.7. The computing device of claim 1, wherein to enforce thecontainer-enclave policy comprises to control access to a hardwarefunction of a hardware security module of the computing device based onthe virtualization context of the application enclave.
 8. The computingdevice of claim 1, wherein to enforce the container-enclave policycomprises to determine whether the application enclave is launchedwithin a virtual machine monitor or whether the application enclave islaunched within a virtual machine.
 9. The computing device of claim 1,wherein to enforce the container-enclave policy comprises to compare asource address of a creating process of the application enclave to apredetermined source address associated with the application enclave.10. The computing device of claim 9, further comprising a virtualmachine monitor to maintain a list of virtual machine source addressesassociated with virtual machines that are successfully loaded by thecomputing device, wherein the predetermined source address is includedin the list of virtual machine source addresses.
 11. The computingdevice of claim 1, further comprising: a policy manager to securelyprovision the container-enclave policy to the computing device; and avirtual machine monitor to construct the platform services enclave inresponse to secure provisioning of the container-enclave policy, whereinthe platform service enclave includes the container-enclave policy. 12.The computing device of claim 11, further comprising a manageabilityengine, wherein to securely provision the container-enclave policycomprises to receive, by the manageability engine, the container-enclavepolicy from an external orchestrator.
 13. The computing device of claim12, wherein the manageability engine is further to inject the containerenclave policy into an image of the virtual machine monitor.
 14. Amethod for secure virtualization, the method comprising: requesting, bya platform services enclave of a computing device, a local attestationreport from an application enclave of the computing device; generating,by the application enclave, an attestation report using secure enclavesupport of a processor of the computing device, wherein the attestationreport is indicative of a virtualization context of the applicationenclave, wherein the virtualization context is provided byvirtualization support of the processor; and enforcing, by the platformservices enclave, a container-enclave policy based on the virtualizationcontext of the application enclave.
 15. The method of claim 14, whereinenforcing the container-enclave policy comprises controlling access to alocal function of the computing device based on the virtualizationcontext of the application enclave.
 16. The method of claim 14, whereinenforcing the container-enclave policy comprises controlling access to ahardware function of a hardware security module of the computing devicebased on the virtualization context of the application enclave.
 17. Themethod of claim 14, wherein enforcing the container-enclave policycomprises determining whether the application enclave is launched withina virtual machine monitor or whether the application enclave is launchedwithin a virtual machine.
 18. The method of claim 14, wherein enforcingthe container-enclave policy comprises comparing a source address of acreating process of the application enclave to a predetermined sourceaddress associated with the application enclave.
 19. The method of claim14, further comprising: securely provisioning, by the computing device,the container-enclave policy to the computing device; and constructing,by the computing device, the platform services enclave by a virtualmachine monitor of the computing device in response to securelyprovisioning the container-enclave policy, wherein the platform serviceenclave includes the container-enclave policy.
 20. One or morecomputer-readable storage media comprising a plurality of instructionsstored thereon that, after being prepared for execution, cause acomputing device that executes the prepared instructions to: request, bya platform services enclave of the computing device, a local attestationreport from an application enclave of the computing device; generate, bythe application enclave, an attestation report using secure enclavesupport of a processor of the computing device, wherein the attestationreport is indicative of a virtualization context of the applicationenclave, wherein the virtualization context is provided byvirtualization support of the processor; and enforce, by the platformservices enclave, a container-enclave policy based on the virtualizationcontext of the application enclave.